1 link tagged with all of: latency + agent performance + ai agents + inference speed
Links
Martin Alderson walks through what he's learned about speeding up agentic AI workflows, drawing on his web performance background. The core argument is that model inference is often not the biggest bottleneck; tool calls, shell startup, polling loops, and hardware choices frequently matter more.
- Inference has two parts to watch: time to first token (dominated by prefill and provider queueing) and tokens per second. Which one matters depends on whether your agent sends short back-and-forth turns or long outputs, and thinking budget can dwarf both.
- Halving inference time only cuts about 20% off a task if provisioning, tool calls, and suspend/resume stay the same. Tool call traces often reveal slow things like xargs spawning hundreds of thousands of processes or a slow .bashrc (nvm is a common culprit).
- Polling loops with sleep are a red flag; the agent is waiting for something that should push a notification instead. Faster underlying hardware (CPU type, local NVMe versus networked disks) and careful RAM management during suspend/resume also make a measurable difference.
ai agents
agent performance
inference speed
latency